Internet Info 1994 March

home *** CD-ROM | disk | FTP | other *** search

/ Internet Info 1994 March / Internet Info CD-ROM (Walnut Creek) (March 1994).iso / inet / internet-drafts / draft-ietf-x400ops-tbl-dist-part1.txt < prev next >

Wrap

Text File | 1993-07-08 | 69KB | 2,205 lines

INTERNET DRAFT Marko Kaittola FUNET Jul 04, 1993 Expires: January, 1994 Mail based file distribution Part 1: Dialog between two nodes Marko Kaittola FUNET Abstract Mapping between X.400 and Internet mail and X.400 routing are normally done using a table-based approach. In practise tables are normal (ASCII) files. In order to function properly tables must be coordinated carefully. One major problem is the lack of automated procedures. This memo - together with it's companion document - proposes one possible solution. This memo discusses the transactions between two nodes, while the companion document discusses the over-all structure aspects. The same solution can be used to transport binary files. This way it is possible to mirror an entire archive with an e-mail only connectivity. Status of this memo This document is an Internet Draft. Internet Drafts are working documents of the Internet Engineering Task Force (IETF), its Areas, and its Working Groups. Note that other groups may also distribute working documents as Internet Drafts. Internet Drafts are draft documents valid for a maximum of six months. Internet Drafts may be updated, replaced, or obsoleted by other documents at any time. It is not appropriate to use Internet Drafts as reference material or to cite them other than as a ``working draft'' or ``work in progress.'' Please check the 1id-abstracts.txt listing contained in the internet-drafts Shadow Directories on nic.ddn.mil, nnsc.nsf.net, nic.nordu.net, ftp.nisc.sri.com, or munnari.oz.au to learn the current status of any Internet Draft. Kaittola Expires: January, 1994 [Page 1] DRAFT File distribution - dialog DRAFT Distribution of this memo is unlimited. Table of contents Abstract 1 Status of this memo 1 Table of contents 2 1. Introduction 3 1.1. About security 3 1.2. Terminology 4 2. Notation 5 3. Parsing 6 4. Dialog 8 4.1. Ihave 10 4.2. Sendme 13 4.3. Data 15 4.4 List 18 4.5. Ping 19 4.6. Pong 19 4.7. Contents of data 20 5. Coding and checksums 23 5.1. Some background 23 5.2. Calculating the checksum 23 6. Security consideratins 26 References 26 Acknowledgements 26 Author's address 26 Appendix A: BNF summary - initial parsing 28 Appendix B: BNF summary - message structure 29 Appendix C: BNF summary - data message parsing 36 Appendix D: Checksum function by Pasi Ojala 38 Kaittola Expires: January, 1994 [Page 2] DRAFT File distribution - dialog DRAFT 1. Introduction This paper defines a dialog between two nodes. [DST-PICTURE] defines a way how this dialog is used. These two documents should be read together. A third document should also be written. That document [DST-MHS] should explain how the other two documents are applied to the needs of X.400 world. This paper has grown from a real need: mapping and routing information in the international R&D X.400 network must be distributed automatically. The most obvious solution would be to use FTP, but unfortunately this isn't always possible. Indeed, the only common denominator seems to be some kind of e-mail connectivity, so this is what the distribution must be based on. Naturally, other distribution techniques may (and most probably will) be used in addition to this. Using a directory (either dns or X.500) is a natural and attractive alternative. This, however, is not yet realistic in a global scale. This proposal tries to fulfill the following requirements: - Files must be distributed rather quickly. In practice this means some minutes from one node to another. Files should reach even their final destination within a couple of hours. - Transport errors and forged files must be automatically identified (and appropriate actions taken). - Management must be simple and require very little human effort. - Distribution must be based on e-mail. This task is simplified by the fact that files are normally public in a sense that anyone may fetch and see them. 1.1. About security This memo defines a secure way for two nodes to communicate together. The over-all structure is discussed in the companion document [DST-PICTURE]. Security is based on a three-phase dialog. The first phase in the initiation of the dialog. It is possible to fake an initiation message ("Ihave message"), but doing so doesn't compromise the security. Kaittola Expires: January, 1994 [Page 3] DRAFT File distribution - dialog DRAFT In the second thase the client send a key to the server. It is expected that although the source of an e-mail is impossible to know the target can be known. Thus the client can be sure that the key is sent to a real server that has been pre-configured on it's database. In the third phase the server respondes. The authentity of the message is checked using the key defined in the second phase. Neither line tapping nor system security are considered. It is expected that if one can listen to the traffic on a line or become a super user one can break the file distribution any way he likes. Doing the key management as part of the dialog simplifies it greatly. 1.2. Terminology A client is a node that receives files and a notification about new files. A server is a node that sends files and a notification about new files. If there is a bi-directional link between any two nodes they can both be clients and servers to each other. Terms client and server are relative to a particular pair of nodes. Kaittola Expires: January, 1994 [Page 4] DRAFT File distribution - dialog DRAFT 2. Notation | choice [] optional <CR> end of line (it can be Carriage Return or Line Feed or anything, system specific) () group <name> named rule * repeated zero or more times *n repeated at most n times m*n repeated between m and n times m* repeated at least m times -- start of comment <l> letter (a-z), case-insensitive <d> digit (0-9) <ldh> letter (a-z), digit (0-9) or hyphen ("-"), case-insensitive <SP> white space (ie. space and tab) <S> space (" ") <any> any single ASCII character except <CR> <(> open parenthis <)> close parenthis <[> open brace <]> close brace anything else literal Implementation must be case insensitive with one exeption: file (and directory) names are case sensitive. Kaittola Expires: January, 1994 [Page 5] DRAFT File distribution - dialog DRAFT 3. Parsing This chapter explains how the parsing of a message is done. Although it is logically done in multiple phases there is no reason why all of these logical phases couldn't be in practice combined into one run. In general a mail message consists of headers and a body. (And an envelope, too. But as an envelope bolongs to MTAs and not UAs it can safely be ignored in this context.) Parsing starts by stripping the headers away. In the second phase trailing white spaces will be removed. In the third phase comments will be removed. A comment is a line that starts with a hash (#), or formally <comment> ::= # <any>* <CR> -- A line starting with a "#" is -- a comment. In the fourth phase empty lines will be removed. In the fifth phase the folding of lines will be undone. A logical line may be folded into multiple physical lines. In practise this will occur if the original (logical) line is considered to be too long. A folded line is formally defined like this: <folded-line> ::= <any>1* ( \ <CR> <SP>1* <any>1* )1* <CR> -- Note that there are no -- insignificant spaces before -- the backslash. If there is a space -- it is significant. Spaces in front -- of the continuation line(s) are -- insignificant and will be removed -- while putting the physical lines -- together. For example: This is an \ example on \ how li\ nes can be folded. produces This is an example on how lines can be folded. Kaittola Expires: January, 1994 [Page 6] DRAFT File distribution - dialog DRAFT As can be seen, a line can be folded even in the middle of a word. Finally, when all of this is done the "real" parsing starts (in the sixth phase). This is defined in chapter four. Please note that chapter four only deals with the sixth phase and the logical lines. Kaittola Expires: January, 1994 [Page 7] DRAFT File distribution - dialog DRAFT 4. Dialog There are normally three phases in a normal ("ihave-sendme-data_or_command") dialog, but it can also start from the phase 2. Phases 2 and 3 can also be replased by using FTP. Doing the dialog in three phases helps to keep it secure, but it also makes the distribution on a large network (relatively) fast and it saves network bandwith. Graphically it can be presented as in the picture 1: SERVER | | CLIENT | | Send a notification |- | | \ | | 1 | IHAVE | \ | | -| Receive a notification | | | -| Send a request | / | | 2 | SENDME | / | Receive a request |- | | | Send files or |- | command | \ | | 3 | DATA or COMMAND | \ | | -| Receive files or | | command | | Picture 1 Phase 1: The server sends a message to a client informing it about the availability of new files or commands. There is no authentication in this phase, although the client may try to see if the notification seems to come from an approved source. Phase 2: The client sends a request to the server. In this request the client specifies a key that the server must use when it replies. The client also stores the key in its local data base. Kaittola Expires: January, 1994 [Page 8] DRAFT File distribution - dialog DRAFT Phase 3: The server responds to the client. Usually this means sending the requested files, but occasionally the server may tell that the files are not available. If the client is requesting a command and it is available, the server responds with the requested command if it is available. When the client receives a reply it checks the validity of that submission based on the key specified in phase 2. There are also variations of the dialog ("ping-pong" and "list-data"). These are presented in the picture 2. SERVER | | CLIENT | | | -| Send a ping message | / | | 1 | PING | / | Receive a ping message |- | | | Send a ping reply |- | | \ | | 2 | PONG | \ | | _| Receive a ping reply | | | | | | | -| Send a list request | / | | 1 | LIST | / | Receive a list request |- | | | Send a listing |- | | \ | | 2 | DATA LIST | \ | | -| Receive a listing | | Picture 2 Phase 1 (ping): The client sends a ping message for test purposes. Phase 2 (pong): The server responds with a pong message. There is no authorization in this dialog Kaittola Expires: January, 1994 [Page 9] DRAFT File distribution - dialog DRAFT (but naturally there is validation). Phase 1 (list): The client sends a list message to receive a listing of available files. Phase 2 (data): The server responds with a data message. This message contains the required listing or an error message. As in a pong message, there is no authorization but there is validation. Doing the distribution in one phase (just sending files) has the following important drawbacks: - If there are loops on a distribution a client will receive multiple copies of the same file. If the file is big this is serious wasting of capasity. - Security must be based on keys that have been set up by hand and that are managed by hand. This is an administrative headache. - It is not possible to request a missing file. Doing the distribution in two phase (just request a file and wait for it) has the following important drawbacks: - It is a slow way to distribute information. - If there are loops on a distribution a client will receive multiple copies of the same file. If the file is big this is serious wasting of capasity. - It isn't possible to fast send a file to replase an incorrect version of it. 4.1. Ihave The server sends an IHAVE message to inform the recipient(s) about the file(s) or command(s) it has received or made available. The following information will be present: - file name - version number - name of distributor It should be noticed that although file names and directory names are case sensitive not all systems support this. The following conventions are strongly suggested: - File names are written in small letters. - Directory names are written in capital letters. It should be noted that a special directory "Cmds" with it's Kaittola Expires: January, 1994 [Page 10] DRAFT File distribution - dialog DRAFT subdirectories is reserved to transmit commands. All of it's subdirectories must be written in capital letters. Naturally it is a local matter on how files and directories are stored locally. For example: # Directory/name IHAVE: FILE TXT COSINE-MHS/mapping-1 # Version number is HHMMSS-DDMMYY VERSION: 930317-121303 FTP: nic.switch.ch:/cosine-mhs/mappings/mapping-1;\ anonymous;user@domain # RFC-addr between a pair of "<>", OR-addr using /-notation. IAM: <cosine-server@cosine-mhs.switch.ch> or # Command IHAVE: CMD Cmds/COSINE-MHS/new-file VERSION: 930424-020127 IAM: /S=cosine-server/OU=cosine-mhs/O=switch/PRMD=SWITCH/\ ADMD=ARCOM/C=CH/ Formal definition: <I-msg> ::= -- IHAVE message ( <I-line> <V-line> [<FTP-line>] )1* <IAM-line> <I-line> ::= -- Commands are given in a special -- file IHAVE: <SP>* ( FILE ( TXT | BINARY ) | CMD ) <SP>* <file-name> <CR> <file-name> ::= <directory>* <filename> -- File name is a symbolic name, -- although it looks like a Unix -- file name. <directory> ::= <l> (<ldh>|_)*14 / -- for example: "COSINE-MHS/" -- This is case-sensitive, but it -- is suggested that capital letters -- are used for a directory whenever -- possible. A special directory -- name "Cmds" (and it's -- subdirectories) are reserved for Kaittola Expires: January, 1994 [Page 11] DRAFT File distribution - dialog DRAFT -- commands. It is explicitly -- forbidden to store any (normal) -- files there. <filename> ::= <l> (<ldh>|_)*14 [ . ( <ldh> | _ )1*14 ] -- For example: "mapping-1" or -- "a_long_file.extension". This is -- case-sensitive, but small -- letters are suggested for a file -- name whenever possible. <V-line> ::= VERSION: <SP>* <vers-number> <CR> <vers-number> ::= <d>6*6 - <d>6*6 -- This must be YYMMDD-hhmmss. -- No timezone is specified. It is -- assumed to be local to the -- node that issues the version -- number. <FTP-line> ::= FTP: <SP>* <host> : <ftp-file> <SP>* ; <SP>* <user> <SP>* ; <SP>* <pass> <CR> -- FTP is one more option for getting -- a file. This line doesn't have to -- be present. Even if it is present -- also normal method (mail) must be -- available. <host> ::= ( <host-component> ( . <host-component> )1* ) | <[> <d>1*3 ( . <d>1*3 )3*3 <]> -- Look at [RFC-952] and -- [RFC-1123] to see the source of -- this definition. <host-component> ::= ( <l> | <d> ) <ldh>1* <ftp-file> ::= -- The name of the file to be get <any>1* -- using FTP. Even a semicolon may -- be part of it. <user> ::= -- Username to be given. <ld>1* <pass> ::= -- Password to be given. <any>1* -- there are two reserved passwords: -- "user@domain" is intended for Kaittola Expires: January, 1994 [Page 12] DRAFT File distribution - dialog DRAFT -- those anonymous FTP servers that -- require a password consisting of -- real username and domainname to -- be used. It should be replased -- with actual data. "SECRET" means -- that password has to be known by -- some other means. <IAM-line> ::= IAM: <SP>* <addr> <CR> <addr> ::= <RFC-addr> | <OR-addr> | ( <RFC-addr> <SP>* <OR-addr> ) -- Also some other kind of address -- could be used. However, it must -- not start or end either with "//" -- nor with "<>". -- Listing both RFC-address and OR -- address might make sense as the -- sender doesn't always know -- which one is more appropriate -- form. <RFC-addr> ::= -- Normal, valid RFC-822 address -- enclosed in "<>" pair. <OR-addr> ::= -- Normal valid OR address using -- "/ notation" not enclosed in -- "<>" pair. 4.2. Sendme In SENDME message the recipient asks for one or more files or commands. A key is given in the request. It is possible to request either a specific version or the latest version. For example: SENDME: FILE COSINE-MHS/mapping-1 # VERSION is either # newest for newest available version # ihave # I have version #; send me the newest if it # isn't this (or smaller) # # send me version # VERSION: newest # Don't bother compressing before sending COMPRESSION: NONE # Data size must not be more that 60 kb in a message. Kaittola Expires: January, 1994 [Page 13] DRAFT File distribution - dialog DRAFT # If there is more data then it must be split into moltiple # messages. MAXSIZE: 60 IAM: <cosine-user@funet.fi> KEY: 1234567890abcdefghij SERIAL: 123 Formal definition: <S-msg> ::= -- SENDME message ( <S-line> <V-line> <CMP-line> )1* <MAX-line> <IAM-line> <K-line> <E-line> <S-line> ::= -- SENDME line SENDME: <SP>* ( FILE | CMD ) <CR> <SP>* <file-name> <CR> <CMP-line> ::= COMPRESSION: <SP>* NONE | ( IS <SP>* <compression> ) | ( CAN <SP>* <compression> [ ; <SP>* <compression> ] ) <CR> -- In sendme-message compression can -- be either NONE or it can be a -- list of possible compressions -- (CAN alternative). In data- -- message it can be either NONE -- or the name of used compression -- (IS-alternative). The idea is -- that when a message is being -- requested it is possible to -- specify all of the possible -- compressions. The server may -- pick one of them, or it may use -- no compression at all. <compression> ::= <ldh>1*15 -- The name of the used compression. -- This can either be bilaterally -- used name or a name that is -- registered somewhere (IANA?). -- The idea is to compress the -- data so that the transport takes -- less time. A good candidate is -- GNU-zip. <MAX-line> ::= Kaittola Expires: January, 1994 [Page 14] DRAFT File distribution - dialog DRAFT MAXSIZE: <SP>* <d>1* <CR> -- This is the maximum size (in kb) -- of data one message may contain. -- a <CR> is counted as two bytes. -- If it's value is zero then there -- is no maximum size set. A server -- may use smaller limit that what -- is set in here, but it must not -- send a message with data part -- bigger than the requested maximum -- size. (The actual size of the -- message is more than the size of -- data. However, the size of the -- data is a dominant factor.) <K-line> ::= -- KEY line KEY: <SP>* <key> <CR> <key> ::= -- signature key <ldh>10*20 <E-line> ::= -- SERIAL line SERIAL: <SP>* <serial> <CR> <serial> ::= -- serial number <d>1*10 -- Serial number consists of -- up to ten digits. It is -- intended that whenever a -- new request is sent the -- serial number is -- incremented by one. -- However, implementation -- may choose to use it -- differently. 4.3. Data The distributor sends the file(s) in a DATA message to the recipient. For example: DATA: FILE TXT COSINE-MHS/mapping-1 VERSION: 121303-170393 PATH: <cosine-server@cosine-mhs.switch.ch> COMPRESSION: NONE CHECK: 123 USED PART: 1 of 3 ---------- start cosine-mhs/mapping-1 ---------- <data> Kaittola Expires: January, 1994 [Page 15] DRAFT File distribution - dialog DRAFT ---------- end cosine-mhs/mapping-1 ---------- IAM: <cosine-server@cosine-mhs.switch.ch> KEY: 1234567890abcdefghij SERIAL: 123 REPLY: + Positive Formal definition: <D-msg> ::= -- DATA message ( <D-line> -- If this block is missing <V-line> -- then reply at K-line must <P-line>1* -- be set to negative. <CMP-line> <C-line> <PART-line> <start-separator> <file> <end-separator> )* <IAM-line> <K-line> <E-line> <R-line> <D-line> ::= -- DATA line DATA: <SP>* ( FILE ( TXT | BINARY ) | CMD | LIST [RECURSIVE] ) <SP>* <file-name> <CR> <P-line> ::= -- PATH line PATH: <SP>* ( IGNORE | <addr> ) <CR> -- There can be more than one PATH -- line. Basically, IAM-line for -- every node that the message have -- passed through should be listed. -- New line will always be placed -- before any other PATH line. -- If there are more than one PTH -- line the last line can contain -- a keyword "IGNORE", which means -- that one or more PATH line has -- been removed. This is intended -- for situations where the actual -- path a file has taken isn't so -- important to know. <PART-line> ::= PART: <SP>* <d>1* <SP>*of <SP>*<d>1* <CR> -- This is needed for multipart -- messages. (Multipart messages -- are needed if such a big file -- has been requested that it can't -- be sent in one message.) If -- message isn't multipart then -- PART: 1 of 1 is used. Kaittola Expires: January, 1994 [Page 16] DRAFT File distribution - dialog DRAFT <start-separator> ::= ---------- <S> start <S> <file-name> <S> ---------- -- Although spacing is defined to be -- strictly like this, when -- receiving a message the amount of -- spaces must be treated as being -- insignificant. The same is true -- for the <end-separator> as well. <file> ::= -- Base64 encoded file to be -- transmitted; As Base 64 doesn't -- use a hyphen there is no danger -- for confusion. -- In addition to Base64 a Hamming -- coding can be used to calculate -- checksums for each line. The -- coding is adjusted to notice if -- lines are duplicated or lost. <end-separator> ::= ---------- <S> <S> end <S> <file-name> <S> <S> ---------- -- Look at the comment for the -- <start-separator>. <C-line> ::= CHECK: <SP>* <check-sum> ( USED | NONE ) <CR> -- If USED then Hamming coding has -- been used to calculate a checksum -- for every line of data. <check-sum> ::= -- Checksum is simply the number of -- data lines (or 33+2 bytes blocks -- if Hamming coding is used). -- It is used to find out if there -- are any lines missing (or -- duplicated). If Hamming coding -- is used, this is used to detect -- blocks that are missing from the -- end. <R-line> ::= -- REPLY line REPLY: <SP>* <reply> <SP>* <explanation> <CR> <reply> ::= -- Status of reply +|- -- Plus or minus, positive or -- negative <explanation> ::= -- A free-form explanation, can be Kaittola Expires: January, 1994 [Page 17] DRAFT File distribution - dialog DRAFT (<ldh>|<SP>)* -- long if line continuation is -- used. If it contains the string -- "\n" it is to be understood as -- an end of line. On of the following explanations must be present at <explanation>: - Positive. Requested file(s) are sent. - Validition failure. A file may or may not exist locally, but the recipient is not served. - File doesn't exist. The requested file is not available. - Too new version. File exists, but version being requested is too new (doesn't exist). - Version not available. Newer version exists locally, but requested version doesn't. - Incorrect request. Something else went wrong. If the requested files are sent the reply status is set to positive, othervise it is set to negative. Answer will contain the information specified above (for logging purposes) and possibly some locally defined explanation. For every request that fail a separate message is required. 4.4 List The client requests a listing of files by sending a LIST message. An example follows. # Directory LIST: COSINE-MHS/ TMP/listing # Data compression (if used) COMPRESSION: NONE # Max size of data in reply MAXSIZE: 60 # RFC-addr between a pair of "<>", OR-addr using /-notation. IAM: <cosine-server@cosine-mhs.switch.ch> # Key KEY: 1234567890abcdefghij # Serial SERIAL: 123 The reply to a LIST message is not an IHAVE message. The reply is generally not intended to be used to request files, although it could be used in that way, too. Kaittola Expires: January, 1994 [Page 18] DRAFT File distribution - dialog DRAFT Formal definition <L-msg> ::= -- LIST message <L-line> <CMP-line> <MAX-line> <IAM-line> <K-line> <E-line> <L-line> ::= LIST: <directory> <SP>1* <file-name> [<SP>1* RECURSIVE] <CR> -- Request either a list of files in -- a directory or a recursive -- listing. The result is sent back -- in <file-name>. 4.5. Ping A ping message is used to test connectivity to the server. The server is expected to reply with a PONG message. For example: PING IAM: <tester@funet.fi> KEY: 1234567890absdefghij SERIAL: 123 Formal definition: <PING-msg> ::= <PING-line> <IAM-line> <K-line> <E-line> <PING-line> ::= PING <CR> 4.6. Pong A pong message is a reply to a ping message. For example: PONG IAM: <cosine-server@cosine-mhs.switch.ch> KEY: 1234567890absdefghij SERIAL: 123 Kaittola Expires: January, 1994 [Page 19] DRAFT File distribution - dialog DRAFT GREETING: Greetings from MHS coordination server! Formal definition: <PONG-msg> ::= <PONG-line> <IAM-line> <K-line> <E-line> <G-line> <PONG-line> ::= PONG <CR> <G-line> ::= GREETING: <SP>* <any>* <CR> -- Greeting is an informal text -- that can be presented to the -- human that uses the ping client -- to test the server. -- Line continuation is to be used -- if all of the text can't be fit -- into one line. Use "\n" to -- represent a line break. 4.7. Contents of data Normally data message carries a (set of) file(s) that can be either text files or binary files. These are encoded using Base64 as defined in [RFC-1341] (pages 17-19). Commands and listings are also carried as normal files. All these cases will be distinquished based on <D-line> where keywords "FILE", "CMD" and "LIST" are used. A keyword "FILE" will be folloved by "TXT" or "BINARY" to describe the type of the file. A command file could look like this: SOURCE: <cosine-server@cosine-mhs.switch.ch> (MHS server) APPROVED-BY: <staff@cosine-mhs.switch.ch> (MHS team) CREATE: cosine-mhs/mapping-notes-1 CREATE: cosine-mhs/mapping-notes-2 DELETE: cosine-mhs/mapping-notes Formally this can be defined like this: <CMD-file> ::= <SOURCE-line> <A-line>1* Kaittola Expires: January, 1994 [Page 20] DRAFT File distribution - dialog DRAFT <CMD-line>1* <SOURCE-line> ::= SOURCE: <SP>* <addr> <SP>* <(> <explanation> <)> <CR> -- This is the source of this -- command file. It includes the -- e-mail address of the source as -- well as a human readable name for -- it. <explanation> ::= -- This is the human readable name <any>1* -- associated with an e-mail -- address. <A-line> ::= APPROVED-BY: <SP>* <addr> <SP>* <(> <explanation> <)> <CR> -- The syntax is very much like in -- <SOURCE-line>. However, they have -- different semantic meanings. -- A <SOURCE-line> tells who -- originally created the command. -- A new <A-line> is created every -- time a message is checked by -- intermediate nodes. Checking (by -- human staff) is mandatory when -- a command for creating or -- deleting a file is sent. It is -- strongly suggested that checking -- will be carried out whenever -- commands are executed. Transit -- nodes can do checking but they -- don't have to. <CMD-line> ::= CREATE | ( DELETE [ <SP>1 RECURSIVE ] ) : ( FILE <SP>* <file-name> ) | ( DIR <SP>* <directory> ) <CR> -- CREATE is used to create either -- a new file or a new directory. -- DELETE is used to delete a file -- or a directory. Unless RECURSIVE -- is specified a directory to be -- deleted must be empty. When sending a listing in a file it looks either like this (normal listing): [cosine-mhs/mapings] [FILE] mapping-1 [FILE] mapping-2 Kaittola Expires: January, 1994 [Page 21] DRAFT File distribution - dialog DRAFT [FILE] mapping-gate [DIR] old [FILE] info [DIR] tools Or it looks like this (recursive listing) [cosine-mhs/mapings] [FILE] mapping-1 [FILE] mapping-2 [FILE] mapping-gate [DIR] old [FILE] mapping-1 [FILE] mapping-2 [FILE] mapping-gate [RID] [FILE] info [DIR] tools [FILE] generate [DIR] data [FILE] countries [RID] [FILE] check [RID] As can be seen, the level of indentation is used to visually indicate which files belong to which directory. Formal definition is like this: <LISTING-file> ::= <N-line> <LIST-line>* <LIST-line> ::= (<S> <S>) * ( <[> ( FILE | DIR ) <]> <SP>* <file-name> ) | ( <[> RID <]> ) <CR> -- Spaces at the beginning of lines -- are used to denote the directory -- structure. They are not -- interpreted while parsing a -- message but used to help a human -- reader understand the structure. -- A directory is started with a -- [DIR] and closed with a [RID]. -- A sophisticated user agent could -- hide [RID] lines from a human, as -- they are only intended to ease -- the parsing. [RID]-lines are not -- needed for non-recursive message. Kaittola Expires: January, 1994 [Page 22] DRAFT File distribution - dialog DRAFT 5. Coding and checksums When data is transmitted it is always encoded, either using normal Base64 as defined in [RFC-1341], or using modified Base64. When using the normal Base64 a line lenght of 76 is to be used when sending a message. However, when receiving a message an arbitrary (and possibly varying) line lenght is to be accepted. The modified Base64 is actually a mixture of Base64 and a Hamming code. The Hamming coding is used for error detection. It could be used for error correction as well, but it is expected that errors are rare but severe, so when an error is found the data is requested to be sent again. 5.1. Some background Using only the normal Base64 is defined well enough in [RFC-1341]. The checksum code used is a (91,88) base-9 Hamming code. This means that three base-9 symbols are generated for every 88 base-9 symbols. One base-9 symbol can fully represent three bits, thus 33 bytes are processed to generate the checksum. Coding those 33 bytes to Base64 symbols makes 44 symbols. Checksum (3 base-9 symbols) is represented with 2 Base64 symbols, thus encoding 33 bytes of original data totals up to a block of 46 Base64 symbols. If the checksum is used a line length of 46 is to be used when encoding data. As usual, any line lenght is to be accepted when decoding data. All arithmetics are performed in remainder class 9. An analysis of reminder class 9 is not provided here, but for example 2 * 7 = (14 modulo 9) = 5 8 + 1 = (9 modulo 9) = 0 The input stream is processed 33 bytes at a time. If there are less than 33 bytes left to process the block is zero-padded for the checksum calculation. This padding is internal to the checksum calculation. 5.2. Calculating the checksum When calculating the checksum the data is handled in Kaittola Expires: January, 1994 [Page 23] DRAFT File distribution - dialog DRAFT three-byte (or 24 bits) groups, like this: Bytes 000000001111111122222222 Base64 000000111111222222333333 Base-9 777666555444333222111000 Three bytes can be split into four Base64 or eight three-bit groups that are interprented as base-9 symbols. 33 bytes generate 11 3-byte groups, 44 base64 symbols and 88 base-9 symbols. The checksum calculation is a simple matrix multiplication. The 88-element base-9 vector is multiplicated with a 3x88-element matrix and the result is a 3-element vector. Because all calculations are done in the remainder class 9, only values from zero to eight are present in the matrix and the results will also be from zero to eight. The reversed order of the base-9 symbols is selected to help to develop as efficient implementation as possible. To generate a checksum vector c one multiplies data vector d with a generator matrix G this way: c = d * G The checksum vector c is then converted into two Base64 symbols as follows: Checksum 000011112222 (byte positions) Base64 000000111111 Four bits are needed to represent one base-9 symbol. Three base-9 symbols total up to 12 bits, which can be represented in two Base64 symbols. The generator matrix G is: Rows 1-22 Rows 23-44 Rows 45-66 Rows 67-88 0 1 1 1 1 4 1 3 8 1 6 3 0 1 2 1 1 5 1 4 0 1 6 4 0 1 3 1 1 6 1 4 1 1 6 5 0 1 4 1 1 7 1 4 2 1 6 6 0 1 5 1 1 8 1 4 3 1 6 7 0 1 6 1 2 0 1 4 4 1 6 8 0 1 7 1 2 1 1 4 5 1 7 0 0 1 8 1 2 2 1 4 6 1 7 1 0 3 1 1 2 3 1 4 7 1 7 2 0 3 2 1 2 4 1 4 8 1 7 3 1 0 1 1 2 5 1 5 0 1 7 4 Kaittola Expires: January, 1994 [Page 24] DRAFT File distribution - dialog DRAFT 1 0 2 1 2 6 1 5 1 1 7 5 1 0 3 1 2 7 1 5 2 1 7 6 1 0 4 1 2 8 1 5 3 1 7 7 1 0 5 1 3 0 1 5 4 1 7 8 1 0 6 1 3 1 1 5 5 1 8 0 1 0 7 1 3 2 1 5 6 1 8 1 1 0 8 1 3 3 1 5 7 1 8 2 1 1 0 1 3 4 1 5 8 1 8 3 1 1 1 1 3 5 1 6 0 1 8 4 1 1 2 1 3 6 1 6 1 1 8 5 1 1 3 1 3 7 1 6 2 1 8 6 However, calculating the checksum in this way is only the first stage: at a second stage the checksum of the previous line (or a zero-vector if the first line is being processed) is added to the checksum vector. This addition is done in reminder class 9 as a vector operation, so there won't be any problems with an over-flow. The <check-sum> that is present in the <C-line> is the number of the 33 bytes blocks, or (if no Hamming coding is used) the number of Base64 lines. Kaittola Expires: January, 1994 [Page 25] DRAFT File distribution - dialog DRAFT 6. Security consideratins Security is based on keys that are sent with the request and copied in a reply. This gives a protection against forged messages. It doesn't work if an ethernet is tapped or some relaying machine is cracked. However, this is considered to be such an extreme situation that such a cracker could in any case cause a great deal of trouble. References [RFC-952] RFC 952 (DOD Internet Host Table Specification) [RFC-1123] RFC 1123 (Requirements for Internet Hosts -- Application and Support) [RFC-1341] RFC 1341 (MIME (Multipurpose Internet Mail Extensions)) [DST-STRUCTURE] Mail based file distribution - Part 2: Over-all structure [DST-MHS] One more companion document to be written (informational) Acknowledgements Various peoples have contributed on this document. It is impossible to list everyone here. However, I'd like to give special thanks to the following: Urs Eppenberger, Allan Cargille, Harald Tveit Alvestrand, Paul Andre Pays and Jim Romaquera from RARE WG1 / RARE WG-MSG / COSINE MHS managers / IETF X.400 ops have contributed and kept me going. Pasi Ojala wrote the first implementation while I wrote this paper. He also suggested many improvements. He developed the approach used in Hamming coding. Keijo Ruohonen helped with the Hamming coding. Author's address Marko Kaittola FUNET c/o Tampere University of Technology Software Systems Laboratory P.O. Box 553 SF-33101 Tampere Finland Kaittola Expires: January, 1994 [Page 26] DRAFT File distribution - dialog DRAFT E-mail: Marko.Kaittola@funet.fi G=Marko; S=Kaittola; O=funet; A=fumail; C=fi; Kaittola Expires: January, 1994 [Page 27] DRAFT File distribution - dialog DRAFT Appendix A: BNF summary - initial parsing <comment> ::= # <any>* <CR> -- A line starting with a "#" is -- a comment. <folded-line> ::= <any>1* ( \ <CR> <SP>1* <any>1* )1* <CR> -- Note that there are no -- insignificant spaces before -- the backslash. If there is a space -- it is significant. Spaces in front -- of the continuation line(s) are -- insignificant and will be removed -- while putting the physical lines -- together. Kaittola Expires: January, 1994 [Page 28] DRAFT File distribution - dialog DRAFT Appendix B: BNF summary - message structure <addr> ::= <RFC-addr> | <OR-addr> | ( <RFC-addr> <SP>* <OR-addr> ) -- Also some other kind of address -- could be used. However, it must -- not start or end either with "//" -- nor with "<>". -- Listing both RFC-address and OR -- address might make sense as the -- sender doesn't always know -- which one is more appropriate -- form. <C-line> ::= CHECK: <SP>* <check-sum> ( USED | NONE ) <CR> -- If USED then Hamming coding has -- been used to calculate a checksum -- for every line of data. <check-sum> ::= -- Checksum is simply the number of -- data lines (or 33+2 bytes blocks -- if Hamming coding is used). -- It is used to find out if there -- are any lines missing (or -- duplicated). If Hamming coding -- is used, this is used to detect -- blocks that are missing from the -- end. <CMP-line> ::= COMPRESSION: <SP>* NONE | ( IS <SP>* <compression> ) | ( CAN <SP>* <compression> [ ; <SP>* <compression> ] ) <CR> -- In sendme-message compression can -- be either NONE or it can be a -- list of possible compressions -- (CAN alternative). In data- -- message it can be either NONE -- or the name of used compression -- (IS-alternative). The idea is -- that when a message is being -- requested it is possible to -- specify all of the possible -- compressions. The server may -- pick one of them, or it may use -- no compression at all. Kaittola Expires: January, 1994 [Page 29] DRAFT File distribution - dialog DRAFT <compression> ::= <ldh>1*15 -- The name of the used compression. -- This can either be bilaterally -- used name or a name that is -- registered somewhere (IANA?). -- The idea is to compress the -- data so that the transport takes -- less time. A good candidate is -- GNU-zip. <D-line> ::= -- DATA line DATA: <SP>* ( FILE ( TXT | BINARY ) | CMD | LIST [RECURSIVE] ) <SP>* <file-name> <CR> <D-msg> ::= -- DATA message ( <D-line> -- If this block is missing <V-line> -- then reply at K-line must <P-line>1* -- be set to negative. <CMP-line> <C-line> <PART-line> <start-separator> <file> <end-separator> )* <IAM-line> <K-line> <E-line> <R-line> <directory> ::= <l> (<ldh>|_)*14 / -- for example: "COSINE-MHS/" -- This is case-sensitive, but it -- is suggested that capital letters -- are used for a directory whenever -- possible. A special directory -- name "Cmds" (and it's -- subdirectories) are reserved for -- commands. It is explicitly -- forbidden to store any (normal) -- files there. <E-line> ::= -- SERIAL line SERIAL: <SP>* <serial> <CR> <end-separator> ::= ---------- <S> <S> end <S> <file-name> <S> <S> ---------- -- Look at the comment for the -- <start-separator>. <explanation> ::= -- A free-form explanation, can be (<ldh>|<SP>)* -- long if line continuation is -- used. If it contains the string Kaittola Expires: January, 1994 [Page 30] DRAFT File distribution - dialog DRAFT -- "\n" it is to be understood as -- an end of line. <file> ::= -- Base64 encoded file to be -- transmitted; As Base 64 doesn't -- use a hyphen there is no danger -- for confusion. -- In addition to Base64 a Hamming -- coding can be used to calculate -- checksums for each line. The -- coding is adjusted to notice if -- lines are duplicated or lost. <file-name> ::= <directory>* <filename> -- File name is a symbolic name, -- although it looks like a Unix -- file name. <filename> ::= <l> (<ldh>|_)*14 [ . ( <ldh> | _ )1*14 ] -- For example: "mapping-1" or -- "a_long_file.extension". This is -- case-sensitive, but small -- letters are suggested for a file -- name whenever possible. <ftp-file> ::= -- The name of the file to be get <any>1* -- using FTP. Even a semicolon may -- be part of it. <FTP-line> ::= FTP: <SP>* <host> : <ftp-file> <SP>* ; <SP>* <user> <SP>* ; <SP>* <pass> <CR> -- FTP is one more option for getting -- a file. This line doesn't have to -- be present. Even if it is present -- also normal method (mail) must be -- available. <G-line> ::= GREETING: <SP>* <any>* <CR> -- Greeting is an informal text -- that can be presented to the -- human that uses the ping client -- to test the server. -- Line continuation is to be used -- if all of the text can't be fit -- into one line. Use "\n" to -- represent a line break. Kaittola Expires: January, 1994 [Page 31] DRAFT File distribution - dialog DRAFT <host> ::= ( <host-component> ( . <host-component> )1* ) | <[> <d>1*3 ( . <d>1*3 )3*3 <]> -- Look at [RFC-952] and -- [RFC-1123] to see the source of -- this definition. <host-component> ::= ( <l> | <d> ) <ldh>1* <I-line> ::= -- Commands are given in a special -- file IHAVE: <SP>* ( FILE ( TXT | BINARY ) | CMD ) <SP>* <file-name> <CR> <I-msg> ::= -- IHAVE message ( <I-line> <V-line> [<FTP-line>] )1* <IAM-line> <IAM-line> ::= IAM: <SP>* <addr> <CR> <K-line> ::= -- KEY line KEY: <SP>* <key> <CR> <key> ::= -- signature key <ldh>10*20 <L-line> ::= LIST: <directory> <SP>1* <file-name> [<SP>1* RECURSIVE] <CR> -- Request either a list of files in -- a directory or a recursive -- listing. The result is sent back -- in <file-name>. <L-msg> ::= -- LIST message <L-line> <CMP-line> <MAX-line> <IAM-line> <K-line> <E-line> <MAX-line> ::= MAXSIZE: <SP>* <d>1* <CR> -- This is the maximum size (in kb) -- of data one message may contain. -- a <CR> is counted as two bytes. -- If it's value is zero then there Kaittola Expires: January, 1994 [Page 32] DRAFT File distribution - dialog DRAFT -- is no maximum size set. A server -- may use smaller limit that what -- is set in here, but it must not -- send a message with data part -- bigger than the requested maximum -- size. (The actual size of the -- message is more than the size of -- data. However, the size of the -- data is a dominant factor.) <OR-addr> ::= -- Normal valid OR address using -- "/ notation" not enclosed in -- "<>" pair. <P-line> ::= -- PATH line PATH: <SP>* ( IGNORE | <addr> ) <CR> -- There can be more than one PATH -- line. Basically, IAM-line for -- every node that the message have -- passed through should be listed. -- New line will always be placed -- before any other PATH line. -- If there are more than one PTH -- line the last line can contain -- a keyword "IGNORE", which means -- that one or more PATH line has -- been removed. This is intended -- for situations where the actual -- path a file has taken isn't so -- important to know. <PART-line> ::= PART: <SP>* <d>1* <SP>*of <SP>*<d>1* <CR> -- This is needed for multipart -- messages. (Multipart messages -- are needed if such a big file -- has been requested that it can't -- be sent in one message.) If -- message isn't multipart then -- PART: 1 of 1 is used. <pass> ::= -- Password to be given. <any>1* -- there are two reserved passwords: -- "user@domain" is intended for -- those anonymous FTP servers that -- require a password consisting of -- real username and domainname to -- be used. It should be replased -- with actual data. "SECRET" means -- that password has to be known by Kaittola Expires: January, 1994 [Page 33] DRAFT File distribution - dialog DRAFT -- some other means. <PING-line> ::= PING <CR> <PING-msg> ::= <PING-line> <IAM-line> <K-line> <E-line> <PONG-line> ::= PONG <CR> <PONG-msg> ::= <PONG-line> <IAM-line> <K-line> <E-line> <G-line> <R-line> ::= -- REPLY line REPLY: <SP>* <reply> <SP>* <explanation> <CR> <reply> ::= -- Status of reply +|- -- Plus or minus, positive or -- negative <RFC-addr> ::= -- Normal, valid RFC-822 address -- enclosed in "<>" pair. <S-line> ::= -- SENDME line SENDME: <SP>* ( FILE | CMD ) <CR> <SP>* <file-name> <CR> <S-msg> ::= -- SENDME message ( <S-line> <V-line> <CMP-line> )1* <MAX-line> <IAM-line> <K-line> <E-line> <serial> ::= -- serial number <d>1*10 -- Serial number consists of -- up to ten digits. It is -- intended that whenever a -- new request is sent the -- serial number is -- incremented by one. -- However, implementation Kaittola Expires: January, 1994 [Page 34] DRAFT File distribution - dialog DRAFT -- may choose to use it -- differently. <start-separator> ::= ---------- <S> start <S> <file-name> <S> ---------- -- Although spacing is defined to be -- strictly like this, when -- receiving a message the amount of -- spaces must be treated as being -- insignificant. The same is true -- for the <end-separator> as well. <user> ::= -- Username to be given. <ld>1* <V-line> ::= VERSION: <SP>* <vers-number> <CR> <vers-number> ::= <d>6*6 - <d>6*6 -- This must be YYMMDD-hhmmss. -- No timezone is specified. It is -- assumed to be local to the -- node that issues the version -- number. Kaittola Expires: January, 1994 [Page 35] DRAFT File distribution - dialog DRAFT Appendix C: BNF summary - data message parsing <A-line> ::= APPROVED-BY: <SP>* <addr> <SP>* <(> <explanation> <)> <CR> -- The syntax is very much like in -- <SOURCE-line>. However, they have -- different semantic meanings. -- A <SOURCE-line> tells who -- originally created the command. -- A new <A-line> is created every -- time a message is checked by -- intermediate nodes. Checking (by -- human staff) is mandatory when -- a command for creating or -- deleting a file is sent. It is -- strongly suggested that checking -- will be carried out whenever -- commands are executed. Transit -- nodes can do checking but they -- don't have to. <CMD-file> ::= <SOURCE-line> <A-line>1* <CMD-line>1* <CMD-line> ::= CREATE | ( DELETE [ <SP>1 RECURSIVE ] ) : ( FILE <SP>* <file-name> ) | ( DIR <SP>* <directory> ) <CR> -- CREATE is used to create either -- a new file or a new directory. -- DELETE is used to delete a file -- or a directory. Unless RECURSIVE -- is specified a directory to be -- deleted must be empty. <explanation> ::= -- This is the human readable name <any>1* -- associated with an e-mail -- address. <LIST-line> ::= (<S> <S>) * ( <[> ( FILE | DIR ) <]> <SP>* <file-name> ) | ( <[> RID <]> ) <CR> -- Spaces at the beginning of lines -- are used to denote the directory -- structure. They are not -- interpreted while parsing a -- message but used to help a human -- reader understand the structure. Kaittola Expires: January, 1994 [Page 36] DRAFT File distribution - dialog DRAFT -- A directory is started with a -- [DIR] and closed with a [RID]. -- A sophisticated user agent could -- hide [RID] lines from a human, as -- they are only intended to ease -- the parsing. [RID]-lines are not -- needed for non-recursive message. <LISTING-file> ::= <N-line> <LIST-line>* <SOURCE-line> ::= SOURCE: <SP>* <addr> <SP>* <(> <explanation> <)> <CR> -- This is the source of this -- command file. It includes the -- e-mail address of the source as -- well as a human readable name for -- it. Kaittola Expires: January, 1994 [Page 37] DRAFT File distribution - dialog DRAFT Appendix D: Checksum function by Pasi Ojala long calcsum(size,buf) int size; /* number of elements, 0-11 */ long *buf; /* 24-bit values */ { int i,j,a,index=0; static int cs0=0,cs1=0,cs2=0; /* Initialize to 0 only once */ long value; /* Matrix columns */ static char a0[]={0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1, 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1}; static char a1[]={1,1,1,1,1,1,1,1,3,3,0,0,0,0,0,0,0,0, 1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2, 3,3,3,3,3,3,3,3,3,4,4,4,4,4,4,4,4,4, 5,5,5,5,5,5,5,5,5,6,6,6,6,6,6,6,6,6, 7,7,7,7,7,7,7,7,7,8,8,8,8,8,8,8}; static char a2[]={1,2,3,4,5,6,7,8,1,2,1,2,3,4,5,6,7,8, 0,1,2,3,4,5,6,7,8,0,1,2,3,4,5,6,7,8, 0,1,2,3,4,5,6,7,8,0,1,2,3,4,5,6,7,8, 0,1,2,3,4,5,6,7,8,0,1,2,3,4,5,6,7,8, 0,1,2,3,4,5,6,7,8,0,1,2,3,4,5,6}; for(i=0;i<size;i++) /* Handle all elements */ { value = buf[i]; /* Get the next 24 bits */ for(j=0;j<8;j++) /* Make it 8*3 bits */ { if((a=(value & 0x07))) /* Get 3 bits */ { cs0 += a0[index] * a; /* Matrix */ cs1 += a1[index] * a; /* multiplication */ cs2 += a2[index] * a; } value >>= 3; /* Forget the processed 3 bits */ index++; /* Go to the next row in the matrix */ } } cs0 %= 9; /* Convert to a remainder class */ cs1 %= 9; /* These can be outside of the loop, */ cs2 %= 9; /* because 88*9*8 = 6336, which fits into */ /* integer nicely */ /* return a 12-bit checksum */ return ((cs0<<8) | (cs1<<4) | cs2); } Kaittola Expires: January, 1994 [Page 38]